Psychomotor vigilance testing for persons tasked with monitoring autonomous vehicles

ABSTRACT

Assessing a likelihood of a person experiencing a fatigue event when the person tasked with monitoring a vehicle operating in an autonomous driving mode may include receiving a set of response times for a psychomotor vigilance test administered to the person. The test may include a plurality of trials which involve a person lifting a finger from a user input device. Whether the person passed or failed each trial of the set of trials may be determined. A model trained using data from prior psychomotor vigilance tests administered to the person may be identified for the person. Results of the determinations of whether the person passed or failed each trial of the set of trials may be input into the model in order to determine a value representative of a likelihood of a fatigue event. An intervention response may be initiated based on the value.

BACKGROUND

Autonomous vehicles, such as vehicles which do not require a humandriver when operating in an autonomous driving mode, may be used to aidin the transport of passengers or items from one location to another.Testing of these vehicles typically involves a “test driver” who istasked with monitoring the autonomous vehicle to ensure that the vehicleis operating safely. For instance, a person may be expected to monitorthe vehicle and the vehicle's environment while the vehicle operates inthe autonomous driving mode and to take control of the vehicle if thereis an emergency or other such situation. Supervision of such vehicles isknown to increase a person's susceptibility to fatigue, when it's due tosleep deprivation, poor quality sleep, fatigue induced by the monitoringitself, or the interaction of these contributing sources of fatigue.

Psychomotor vigilance tests (PVTs) may be used to test a person'scurrent state of awareness by way of patterns in his or her reactiontimes. For instance, a PVT may require a person to perform a specificaction, such as pressing a button, as soon as the person recognizes thata particular image or message is displayed or a particular audio isplayed. Current PVTs have demonstrated that a single set of PVTevaluation criteria can be used to predict the future task performanceof a group of fatigued people (such as driving performance). However,individuals may vary greatly in their baseline PVT performance, whichmay make existing PVTs less sensitive to some persons' fatiguedresponses and overly sensitive to other persons' fatigued responses.This, in turn, may result in group-level PVT pass/fail criteria toproduce false negatives for some and false positives for others,respectively.

SUMMARY

One aspect of the disclosure provides a method of assessing a likelihoodof a person experiencing a fatigue event, the person tasked withmonitoring a vehicle operating in an autonomous driving mode. The methodincludes receiving, by one or more server computing devices, a set ofresponse times for a psychomotor vigilance test administered to theperson, the test including a set of trials; determining, by the one ormore server computing devices, whether the person passed or failed eachtrial of the set of trials; identifying, by the one or more servercomputing devices, a model associated with the person, the model beingtrained using data from prior psychomotor vigilance tests administeredto the person; inputting, by the one or more server computing devices,results of the determinations of whether the person passed or failedeach trial of the set of trials into the model in order to determine avalue representative of a likelihood of a fatigue event; and initiating,by the one or more server computing devices, an intervention responsebased on the value.

In one example, the method also includes determining the interventionresponse by comparing the value to one or more threshold values. Inanother example, the method also includes determining a score for thetest based on the determinations of whether the person passed or failedeach trial of the set of trials, and the results include the score. Inthis example, the score represents a passing or failing rate for theperson. In another example, the method also includes inputting date andtime information for the set of trials into the model in order todetermine the value. In another example, the method also includesinputting shift information about a relative point in time for a shiftfor monitoring the vehicle of the person for the test into the model inorder to determine the value. In another example, the method alsoincludes inputting an amount of time since a last break of the personfor the test into the model in order to determine the value. In anotherexample, the method also includes inputting information identifyingwhere a person is with respect to his or her circadian rhythm, for thetest into the model in order to determine the value. In another example,the method also includes determining a threshold amount of time based onwhether the person is left-handed or right-handed, and determiningwhether the person passed or failed each trial of the set of trials isbased on the threshold amount of time. In another example, the methodalso includes determining a threshold amount of time using a model ofhow response times change over time due to muscle fatigue, anddetermining whether the person passed or failed each trial of the set oftrials is based on the threshold amount of time. In another example, themethod also includes determining a threshold amount of time using amodel of how response times change over time for the person, anddetermining whether the person passed or failed each trial of the set oftrials is based on the threshold amount of time. In another example, themethod also includes determining a threshold amount of time using amodel of how response times change over time for a similarly situatedperson, and determining whether the person passed or failed each trialof the set of trials is based on the threshold amount of time. Inanother example, the method also includes determining a threshold amountof time is based on an orientation of a user input device used toadminister the set of trials, and determining whether the person passedor failed each trial of the set of trials is based on the thresholdamount of time. In another example, the method also includes determininga threshold amount of time is based on an orientation of a user inputdevice used to administer the set of trial, and determining whether theperson passed or failed each trial of the set of trials is based on thethreshold amount of time. In another example, the method also includes,once an intervention response is identified, sending one or more signalsto one or more of the computing devices of the vehicle, a PVT systemthat administered the test, and a computing device of a remoteassistance operator in order to initiate the identified interventionresponse. In this example, the signal includes instructions for thetesting system to administer a second test. In addition, the signalincludes instructions for the vehicle to stop operating in theautonomous driving mode. In addition, the signal includes instructionsfor the vehicle to prevent the person from operating the vehicle in amanual driving mode. In addition or alternatively, the signal enablesthe person to communicate with the remote assistance operator.

Another aspect of the disclosure provides a method of training a modelfor estimating a likelihood of a fatigue event for a person tasked withmonitoring a vehicle operating in an autonomous driving mode. The methodincludes accessing, by one or more server computing devices, resultsfrom a plurality of sets of psychomotor vigilance test administered tothe person at different points in time, the results including responsetimes for the person; determining, by the one or more server computingdevices, scores for each of the sets of tests based on the results;accessing, by the one or more server computing devices, information froma remote monitoring system identifying estimations of the person'samount of fatigue at different times; determining, by the one or moreserver computing devices, whether the information indicates that theperson experienced one or more fatigue events; and using, by the one ormore server computing devices, the determined scores and thedetermination of whether the information indicates that the personexperienced one or more fatigue events to train a model individualizedto the person such that when data from a set of tests are input into themodel, the model outputs a value indicative of a likelihood of theperson experiencing a fatigue event.

Another aspect of the disclosure provides a method of administering apsychomotor vigilance test to a person tasked with monitoring a vehicleoperating in an autonomous driving mode. The method includes providing,by one or more processors, for display to the person, a visual stimuluson a touch-sensitive display while the person is resting a finger on thedisplay; determining, by one or more processors, based on feedback fromthe touch-sensitive display, a point in time when the finger was removedfrom the display; measuring, by one or more processors, an amount oftime for the person to remove the finger from the display between whenthe visual stimulus was provided and the point in time; and sending, bythe one or more processors, the amount of time to a remote computingsystem in order to determine a likelihood of a fatigue event while theperson is tasked with monitoring the vehicle.

In one example, each stimulus and amount of time correspond to a singletrial, and wherein the test includes a plurality of trials. In anotherexample, the method also includes providing instructions, by the one ormore processors via the display, to the person to place the finger onthe display prior to providing the visual stimulus.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an example of a PVT system.

FIG. 2 is a functional diagram of an example vehicle in accordance withan exemplary embodiment.

FIG. 3 is an example external view of a vehicle in accordance withaspects of the disclosure.

FIG. 4 is an example pictorial diagram of a system in accordance withaspects of the disclosure.

FIG. 5 is an example block diagram of the system of FIG. 4 .

FIG. 6 is an example flow diagram in accordance with aspects of thedisclosure.

FIG. 7 is an example block diagram representing training of a model inaccordance with aspects of the disclosure.

FIG. 8 is an example flow diagram in accordance with aspects of thedisclosure.

FIG. 9 is an example flow diagram in accordance with aspects of thedisclosure.

DETAILED DESCRIPTION

Overview

The technology relates to psychomotor vigilance test (PVTs) for personswho are tasked with monitoring the driving of a vehicle operating in anautonomous driving mode. For instance, a person may be expected tomonitor the vehicle and the vehicle's environment while the vehicleoperates in the autonomous driving mode and to take control of thevehicle if there is an emergency or other such situation. Supervision ofsuch vehicles is known to increase a person's susceptibility to fatigue,when it's due to sleep deprivation, poor quality sleep, fatigue inducedby the monitoring itself, or the interaction of these contributingsources of fatigue. To test a person's current state of awareness by wayof patterns in his or her reaction times, a PVT may require a person toperform a specific action, such as press or lifting a finger away from abutton or a screen as soon as the person recognizes that a particularimage or message is displayed or a particular audio is played. CurrentPVTs have demonstrated that a single set of PVT evaluation criteria canbe used to predict future task performance (such as drivingperformance). However, individuals may vary greatly in their baselinePVT performance, which may make existing PVTs less sensitive to somepersons' responses and overly sensitive to other persons' responses.This, in turn, may result in group-level PVT pass/fail criteria thatproduce false negatives for some and false positives for others,respectively.

A testing system may include an output device, such as a touchscreenand/or speaker, as well as a user input device. The user input devicemay be connected to a computing device that can measure a time betweenwhen information is displayed on the output device and when a personcompletes a trial of a test, for instance by tapping with a finger onthe touchscreen or alternatively by lifting the finger from thetouchscreen. For instance, the computing device may provide a stimulus,such as displaying information on a screen, and start a timer orcounter. When the person's finger is removed from the touchscreen, thecounter may be used to determine how long it takes for the user inputdevice to register a response of the person for the trial. This maycorrespond to a response time, which in effect accounts for both areaction time (when the person recognizes the stimulus) and the physicalmovement time (the time required for a person to move his or herfinger). The response time may then be compared to some pass/failcriteria such as a threshold amount of time to determine whether theperson has passed or failed the trial.

As noted above, the testing system may be employed within a vehiclehaving an autonomous driving mode. As such, the testing system may beable to communicate with the computing devices of the vehicle as well asvarious remote computing devices either directly or indirectly via thecomputing devices of the vehicle.

To address the aforementioned short-comings related to false positives(or rather failures that should not actually be considered failures),the features described here may reduce error due to counting movementtime as response time and create individualized pass/fail criteria thataccount for error due to differing baseline performance betweenindividuals. For instance, rather than simply monitoring a press of abutton or screen, each trial of a test may be configured as a fingerlift. In this regard, each trial of a test may begin with a person'sfinger rested on a touchscreen or other touch sensitive device.

A person may take a plurality of tests at different times of day duringdifferent shifts. The reaction times may be sent by the testing systemto a remote server computing device. In some instances, a plurality oftrials from a single test may be scored. The scoring may be based on theresponse times.

This information may be combined with other signals. For instance, anin-vehicle or remote monitoring system may be used to determine aperson's fatigue. As another instance, the testing system may include anoption for a person to self-report the state of his or her fatigue onthe same or a different assessment scale.

Results from a particular person's PVTs as well as the other signals maybe used to train a model. The model may be a regression model or astatistical model trained by one or more server computing devices. ThePVT data may include both the dates, times, and point in time relativeto the person's shift that a series of PVTs was performed as well as theresponse times and/or scores. This PVT data may be used as traininginputs. The data from the in-car or remote monitoring system and/or theself-reported data may be used to determine whether a set of PVTscorresponds to a fatigue event (e.g. stop paying attention or evenfalling asleep). These determinations (fatigue event or no fatigueevent) may be used as the training outputs.

The model may be trained such that inputting new PVT data and point withrespect to his or her shift will provide an estimate of a person'scurrent and/or future state of fatigue. The more PVT data and othersignals used to train the model the more accurate the probabilityestimation that the person will have a fatigue event. The model may thenbe associated with the person, for instance using an identifying code orother information, and saved for later retrieval.

When and how often the testing system administers a test may varydepending upon the situation. To administer a test, the vehicle'scomputing devices may send a signal to the testing system when suchsituations occur. The testing system may test a person by administeringa set of trials and evaluating the person's response times for thosetrials as described above. The response times as well as date and timeinformation for the trials of a test, shift information about therelative point in the person's shifts, as well as any of the additionalinformation discussed above (if available) may then be sent to andreceived by the remote server computing device.

The remote server computing device may then use the reaction times todetermine whether the person passed or failed each of the set of trialsof a test. This may also include determining a score for the trials ofthe set. The model associated with the particular person, which as notedabove, was trained using data from prior tests administered to thatperson as well as other signals as discussed above may be identified.The results of the determinations of whether the person passed or failedeach PVT of the set of PVTs or alternatively, the response times, may beinput into the model in order to determine a value representative of alikelihood of a fatigue event.

In order to determine and initiate an intervention response, the valuemay then be compared to a threshold value or plurality of thresholdvalues to determine whether and what type of intervention response totake. Once an intervention response is identified, the server computingdevices may send a signal to the computing devices of the vehicle, thePVT system, and/or a remote assistance operator in order to initiate theidentified intervention response. Various different types ofintervention responses may be employed.

The features described herein may provide for a reliable and effectivesystem for identifying possible fatigue events in persons tasked withmonitoring the driving of a vehicle operating in an autonomous drivingmode. For instance, because the model is trained specific to aparticular person or a similarly situated person, the model may be morelikely to identify outlier situations. Moreover, because the thresholdamount of time used to determine whether a person has passed or failed aparticular PVT can be adjusted to accommodate for the particular personor similarly situated persons, this may further increase theeffectiveness of the system reduce false positives. In other words,threshold amounts of time may be set in a way that accounts forindividual variance, for instance, due to muscle fatigue.

Example Systems

A testing system may include one or more computing devices 110 havingone or more processors 120 and memory 130 storing instructions 132 anddata 134. The memory 130 stores information accessible by the one ormore processors 120, including instructions 132 and data 134 that may beexecuted or otherwise used by the processor 120. The memory 130 may beof any type capable of storing information accessible by the processor,including a computing device-readable medium, or other medium thatstores data that may be read with the aid of an electronic device, suchas a hard-drive, memory card, ROM, RAM, DVD or other optical disks, aswell as other write-capable and read-only memories. Systems and methodsmay include different combinations of the foregoing, whereby differentportions of the instructions and data are stored on different types ofmedia.

The instructions 132 may be any set of instructions to be executeddirectly (such as machine code) or indirectly (such as scripts) by theprocessor. For example, the instructions may be stored as computingdevice code on the computing device-readable medium. In that regard, theterms “instructions” and “programs” may be used interchangeably herein.The instructions may be stored in object code format for directprocessing by the processor, or in any other computing device languageincluding scripts or collections of independent source code modules thatare interpreted on demand or compiled in advance. Functions, methods androutines of the instructions are explained in more detail below.

The data 134 may be retrieved, stored or modified by processor 120 inaccordance with the instructions 132. For instance, although the claimedsubject matter is not limited by any particular data structure, the datamay be stored in computing device registers, in a relational database asa table having a plurality of different fields and records, XMLdocuments or flat files. The data may also be formatted in any computingdevice-readable format.

The one or more processors 120 may be any conventional processors, suchas commercially available CPUs or GPUs. Alternatively, the one or moreprocessors may be a dedicated device such as an ASIC or otherhardware-based processor. Although FIG. 1 functionally illustrates theprocessor, memory, and other elements of computing device 110 as beingwithin the same block, it will be understood by those of ordinary skillin the art that the processor, computing device, or memory may actuallyinclude multiple processors, computing devices, or memories that may ormay not be stored within the same physical housing. For example, memorymay be a hard drive or other storage media located in a housingdifferent from that of computing device 110. Accordingly, references toa processor or computing device will be understood to include referencesto a collection of processors or computing devices or memories that mayor may not operate in parallel.

The computing device 110 may also include an output device 160, such asa touchscreen and/or speaker, as well as a user input device 150, suchas a touchscreen, button or other touch sensitive device. In thisregard, the output device and the user input device may be the samedevice (e.g. a touchscreen). The user input device 150 may beincorporated into or connected to a computing device, such as computingdevice 110, that can measure a time between when information isdisplayed on the output device and when a person completes a trial of atest, for instance by tapping with a finger on the touchscreen oralternatively by lifting the finger from the touchscreen. For instance,the computing device 110 may provide a stimulus, such as displayinginformation on a screen, and start a timer or counter. When the person'sfinger is removed from the touchscreen, the counter may be used todetermine how long it takes for the user input device 150 to register aresponse of the person for the trial. This may correspond to a responsetime, which in effect accounts for both a reaction time (when the personrecognizes the stimulus) and the physical response time (when the personmoves his or her finger). The response time may then be compared to somepass/fail criteria such as a threshold amount of time to determinewhether the person has passed or failed the trial.

As an example, in a single test, there may be as many as 60-70 trialswhich all together may take approximately 3 minutes to complete. Forinstance, the tests may utilize an inter-stimulus-interval (ISA) of 1-4seconds, varying randomly between trials/stimuli. If the ISA were to beset to 1-3 seconds, there would be many more trials within the same test(currently at 3 minutes). Additionally, the test length can be increasedor decreased, which would also increase or decrease the total number oftrials for the test.

The testing system 100 may also include a communications system 140 thatenables the testing system 100 to communicate with other computingdevices. For example, the communication system may include wired and/orwireless connections (such as transmitters and receivers), that enablethe testing system to communicate with other computing devices. As anexample, the communications system may enable the testing system to usevarious protocols including short range communication protocols such asBluetooth, Bluetooth LE, the Internet, World Wide Web, intranets,virtual private networks, wide area networks, local networks, privatenetworks using communication protocols proprietary to one or morecompanies, Ethernet, WiFi and HTTP, and various combinations of theforegoing. Such communication may be facilitated by any device capableof transmitting data to and from other computing devices, such as modemsand wireless interfaces.

As noted above, the testing system may be employed within a vehiclehaving an autonomous driving mode. FIG. 2 is an example block diagram ofa vehicle 200, and FIG. 3 is an example view of the vehicle 200. In thisexample, the vehicle 200 is a vehicle having an autonomous driving modeas well as one or more additional driving modes, such as asemiautonomous or manual driving mode. While certain aspects of thedisclosure are particularly useful in connection with specific types ofvehicles, the vehicle may be any type of vehicle including, but notlimited to, cars, trucks, motorcycles, buses, recreational vehicles,etc.

Turning to FIG. 2 , the computing devices 110 of the testing system maybe in communication with one or more computing devices 210 of thevehicle. The one or more computing devices 210 may include one or moreprocessors 220, memory 230 storing instructions 232 and data 234, andother components typically present in general purpose computing devices.These processors, memory, instructions and data may be configured thesame or similarly to the processors 120, memory 130, instructions 132,and data 134.

In one aspect the computing devices 210 may be part of an autonomouscontrol system capable of communicating with various components of thevehicle in order to control the vehicle in an autonomous driving mode.For example, returning to FIG. 2 , the computing devices 210 may be incommunication with various systems 250, 260, 270 via wired or wirelessconnections. As an example, these systems may correspond to adeceleration system, an acceleration system, a steering system, arouting system for determining a route for the vehicle to follow betweentwo or more locations, and planning system for planning a trajectory, apositioning system, a perception system for detecting objects in thevehicle's environment, etc. which the computing devices can use tocontrol the vehicle 200 in the autonomous and semiautonomous drivingmodes.

The vehicle 200 may also include remote monitoring system 280. Thissystem may include hardware features such as video cameras, microphonesand speakers as well as one or more computing devices and communicationssystem which may be configured the same or similarly to computing device110 and communications system 140. The hardware features may be used toenable a remote operator to “check-in” on a test driver as well as toenable two-way communications between the remote operator and the testdriver.

Turning to FIG. 3 , as an example, the vehicle 200 includes a roof-tophousing 310 and dome housing 312 may include a LIDAR sensor as well asvarious cameras and radar units. In addition, housing 320 located at thefront end of the vehicle 200 and housings 330, 332 on the driver's andpassenger's sides of the vehicle may each store a LIDAR sensor. Forexample, housing 330 is located in front of a driver door 360. Vehicle100 also includes housings 340, 342 for radar units and/or cameras alsolocated on the roof of vehicle 200. Additional radar units and cameras(not shown) may be located at the front and rear ends of vehicle 200and/or on other positions along the roof or roof-top housing 310.

The computing devices 210 may include a communications system 240 whichmay be the same or similar to communications system 140. Thecommunications system may enable the computing devices 210 tocommunicate with other devices remote from the vehicle. In this way,information from the testing system 100 may be sent to remote devices.As such, the testing system 100 may be able to communicate with thecomputing devices 210 of the vehicle as well as various remote computingdevices, such as those computing devices that are a part of theautonomous vehicle service as well as other computing devices, eitherdirectly or indirectly via the computing devices of the vehicle.

FIGS. 4 and 5 are pictorial and functional diagrams, respectively, of anexample system 400 that includes a plurality of computing devices 410,420, 430, 440 and a storage system 450 connected via a network 460.System 400 also includes vehicles 200A, 200B, 200C, 200D, which may beconfigured the same as or similarly to vehicle 200. Although only a fewvehicles and computing devices are depicted for simplicity, a typicalsystem may include significantly more.

As shown in FIG. 4 , each of computing devices 410, 420, 430, 440 mayinclude one or more processors, memory, data and instructions. Suchprocessors, memories, data and instructions may be configured similarlyto one or more processors 120, memory 130, data 134, and instructions132 of computing device 110.

The network 460, and intervening nodes, may include variousconfigurations and protocols including short range communicationprotocols such as Bluetooth, Bluetooth LE, the Internet, World Wide Web,intranets, virtual private networks, wide area networks, local networks,private networks using communication protocols proprietary to one ormore companies, Ethernet, WiFi and HTTP, and various combinations of theforegoing. Again, communication may be facilitated by any device capableof transmitting data to and from other computing devices, such as modemsand wireless interfaces.

In one example, one or more computing devices 410 may include one ormore server computing devices having a plurality of computing devices,e.g., a load balanced server farm, that exchange information withdifferent nodes of a network for the purpose of receiving, processingand transmitting the data to and from other computing devices. Forinstance, one or more computing devices 410 may include one or moreserver computing devices that are capable of communicating withcomputing device 210 of vehicle 200 or a similar computing device ofother vehicles as well as computing devices 420, 430, 440 via thenetwork 460. For example, each of the vehicles 200A, 200B, 200C, 200D,may correspond to vehicle 200 and may be a part of a fleet of vehiclesof the autonomous vehicle service that can be dispatched by servercomputing devices 410 to various locations. In this regard, the servercomputing devices 410 may function (in conjunction with storage system450) as a dispatching system for the autonomous vehicle service whichcan be used to dispatch vehicles such as vehicle 200 and vehicle 200A todifferent locations in order to pick up and drop off passengers. Inaddition, server computing devices 410 may use network 460 to transmitand present information to a person, such as people 422, 432, 442 on adisplay, such as displays 424, 434, 444 of computing devices 420, 430,440. In this regard, computing devices 420, 430, 440 may be consideredclient computing devices.

As shown in FIG. 4 , each client computing device 420, 430, 440 may be apersonal computing device intended for use by a person 422, 432, 442,and have all of the components normally used in connection with apersonal computing device including a one or more processors (e.g., acentral processing unit (CPU)), memory (e.g., RAM and internal harddrives) storing data and instructions, a display such as displays 424,434, 444 (e.g., a monitor having a screen, a touch-screen, a projector,a television, or other device that is operable to display information),and user input devices 426, 436, 446 (e.g., a mouse, keyboard,touchscreen or microphone). The client computing devices may alsoinclude a camera for recording video streams, speakers, a networkinterface device, and all of the components used for connecting theseelements to one another.

Although the client computing devices 420, 430, and 440 may eachcomprise a full-sized personal computing device, they may alternativelycomprise mobile computing devices capable of wirelessly exchanging datawith a server over a network such as the Internet. By way of exampleonly, the client computing devices may include a mobile phone or adevice such as a wireless-enabled PDA, a tablet PC, a wearable computingdevice or system, or a netbook that is capable of obtaining informationvia the Internet or other networks.

Each of the client computing devices may be remote monitoring workstation used by an administrator or remote assistance operator (e.g.people 422, 432, 444) to provide concierge or remote assistance servicesto test drivers of vehicles 200A, 200B, 200C, 200D. For example, arepresentative 442 may use the remote monitoring workstation 440 tocommunicate via a telephone call or audio connection with people throughtheir respective client computing devices or vehicles 200A, 200B, 200C,200D, in order to ensure the safe operation of vehicles 100 and 100A andthe safety of the test drivers as described in further detail below.Although only a few remote monitoring workstation 440 is shown in FIGS.4 and 5 , any number of such work stations may be included in a typicalsystem.

As with memory 130, storage system 450 can be of any type ofcomputerized storage capable of storing information accessible by theserver computing devices 410, such as a hard-drive, memory card, ROM,RAM, DVD, CD-ROM, write-capable, and read-only memories. In addition,storage system 450 may include a distributed storage system where datais stored on a plurality of different storage devices which may bephysically located at the same or different geographic locations.Storage system 450 may be connected to the computing devices via thenetwork 460 as shown in FIGS. 3 and 4 , and/or may be directly connectedto or incorporated into any of the computing devices 410, 420, 430, 440,etc.

The storage system 450 may be configured to store various information.This information may include the status (serving a trip, passengers,etc.), current locations, and expected future locations of the vehiclesof the fleet. The information may also include data associated with aparticular test driver. For instance, each test driver identified in thestorage system may be associated with one or more models (or modelparameter values for use with a generic model for all test drivers),reaction times, scores, dates and times of PVTs, shift information,fatigue information from remote monitoring system(s) and/orself-reported data, as well as additional information as discussedfurther below.

Example Methods

In addition to the operations described above and illustrated in thefigures, various operations will now be described. It should beunderstood that the following operations do not have to be performed inthe precise order described below. Rather, various steps can be handledin a different order or simultaneously, and steps may also be added oromitted.

As noted above, to address the aforementioned short-comings related tofalse positives (or rather failures that should not actually beconsidered failures), the features described here may reduce error dueto counting movement time as response time and create individualizedpass/fail criteria that account for error due to differing baselineperformance between individuals. For instance, rather than simplymonitoring a press of a button or screen, each trial of a test may beconfigured as a finger lift. For example, if a person taking a testholds their finger sometimes at 1 inch and sometimes at 3 inches away,the variation in movement time will be counted as variance in reactiontime even though in both cases the person may have begun to reactequally as fast. In this regard, each trial of a test may begin with aperson's finger placed, rested or otherwise positioned on a touchscreenor other touch sensitive device. Therefore, the initial conditions foreach trial may be identical or nearly identical because movementdistance is held constant over all of the tests and all of the personsto be tested. In other words, variation between the circumstances ofeach trial or across many tests is virtually non-existent with theexception of the characteristics of the persons taking the tests. Inaddition, this arrangement may also be less muscle intensive for somepersons as lifting a finger at rest may be easier than holding a fingerabove an input device for test. In other words, the person can befocused on waiting for the stimulus rather than holding his or herfinger at a particular location above the touchscreen.

A person, or rather, a test driver, may take a plurality of tests (PVTs)at different times of day during different shifts (e.g. the periods oftime during which the test driver is expected to monitor the driving ofthe vehicle in the autonomous driving mode which may also include one ormore breaks in such periods). The reaction times may be sent by thetesting system 100 to a remote server computing device such as servercomputing device 410. This data may be stored, for instance, in thestorage system 450.

FIG. 6 is an example flow diagram 600 for administering a psychomotorvigilance test to a person tasked with monitoring a vehicle operating inan autonomous driving mode which may be performed by one or moreprocessors of one or more computing devices, such as the processors 120of computing devices 110 of the testing system 100. At block 610, avisual stimulus is provided for display on a touch-sensitive displaywhile the person is resting a finger on the display. At block 620, basedon feedback from the touch-sensitive display, a point in time when thefinger was removed from the display is determined. At block 630, anamount of time for the person to remove the finger from the displaybetween when the visual stimulus was provided and the point in time ismeasured. At block 640, the amount of time is provided to a remotecomputing system in order to determine a likelihood of a fatigue eventwhile the person is tasked with monitoring the vehicle. Each stimulusand amount of time correspond to a single trial, and wherein the testincludes a plurality of trials. In addition, the test may also includeproviding instructions to the person to place the finger on the displayprior to providing the visual stimulus.

In some instances, a plurality of trials from a single PVT may be scoredby the server computing devices 410 and/or the computing device 110 andthereafter sent to the server computing devices 410. The scores may alsobe stored in the storage system 450.

The scoring may be determined based on the response times. In oneexample, if a threshold amount of time is 300 milliseconds, and areaction time less than 355 milliseconds, the test driver may beconsidered to have passed. If the response time is greater than 355milliseconds, the test driver may be considered to have failed or tohave a lapse in his or her attention. As another example, if the testdriver's response time is measured within 100 milliseconds of, orbefore, a stimulus for a trial is displayed or played, the test drivermay also be considered to have failed or to have had a lapse. In thisregard, the 100 milliseconds may also identify false response times as100 ms corresponds the transmission limits of a human nervous system. Insituations where false response times or “false starts” are prevalent,the test may be disregarded and administered again or the trial maysimply be identified as a failure. A ratio of the number of passed orfailed trials to the total amount of trials in a test which representsthe passenger's rate of passing or failing for the test may bedetermined. Thus, the ratio may provide a score on a scale of 0 to 1.Other scoring methods may also be used.

This information may be combined with other signals. For instance, theremote monitoring system 280 may be used to determine a test driver'sfatigue. For example, as noted above, the remote monitoring system 280may include one or more video or still cameras which provide a remoteassistance operator, such as people 422, 432, 442, to “check in” on thetest driver while he or she is monitoring a vehicle, such as vehicle200. For instance, the remote assistance operator may identify signalssuch as a test driver “nodding off”, slowly closing his or her eyes, notmonitoring the road with his or her eyes, etc. This same video or cameradata may also be used by an automated system (e.g. an in-vehiclemonitoring system) to look for similar signals. As another instance, thetesting system may include an option for a test driver to self-reportthe state of his or her fatigue on the same or a different assessmentscale, such as entering a value on a scale of 0 to 1, 0 to 10, 0 to 100,etc. This fatigue information may also be stored in the storage system450. As yet another example of other signals, patterns in thetime-to-reset (TTR) may also cause fatigue. In other words, the time ittakes for the driver to put his or her finger back on the display,before each task can increase over time. Thus, fatigue could productlonger TTRs due to lapses in vigilance.

Results from a particular test driver's PVTs as well as the othersignals may be used to train a model to determine parameter values forthat particular test driver. Alternatively, rather than being for aparticular test driver, the model may be trained for a similarlysituated test driver. For example, drivers whose baseline PVTperformance (i.e. scores) is very similar to other drivers may begrouped together, and the results from the PVTs of these drivers as wellas the other signals may be used to train a model. In this regard, themodel may produce a set of parameters effective at detecting fatigue forthat group specifically. The model may be a regression model, neuralnetwork, random forest or any other predictive model that can produce anumerical prediction on a scale. The model may be trained by one or moreserver computing devices, such as the server computing devices 410.

FIG. 7 is an example representation of training a model 710 for aparticular test driver, though similar approaches may be used fortraining a model for similarly situated test drivers. PVT data for theparticular test driver may be used as training inputs 720. The PVT datamay include both the dates, times, and point in time relative to theparticular test driver's shift that a series of PVTs was performed aswell as the response times and/or scores.

The data from the remote monitoring system and/or the self-reported datafor that particular test driver may be used to determine whether a setof PVTs corresponds to a fatigue event (e.g. stop paying attention oreven fall asleep). For instance, PVT data may be correlated (forexample, manually) with the other signals in two groups: those thatcorresponded to a later fatigue event during the same shift and thosethat did not. These determinations (fatigue event or no fatigue event)may be used as the training outputs 730. The training may produceparameter values for the model that are useful for predicting fatigueevents for the particular test driver.

Alternatively, the model may be a statistical model that relates PVTdata with a likelihood of a fatigue event for a particular test driver.This may be most useful when there is not a lot of training data for aparticular test driver.

The model may be trained such that inputting new PVT data and point intime with respect to his or her shift will provide an estimate of a testdriver's current and/or future state of fatigue. For instance, bylooking at reaction times immediately before or right after the start ofa shift, the model may be able to do a better job of predicting a testdriver's future fatigue (e.g. later in the shift). In some instances,the training data may also include additional information such as wherea test driver is with respect to his or her circadian rhythm, time sincethe test driver's last break (or “time on task” corresponding to theduration of time that the test driver has spent monitoring the vehicleuninterrupted) such that this information may also be used as input tothe model to provide an estimate of a test driver's current and/orfuture state of fatigue. This estimate may be a value, for instance on ascale of 0 to 1 which represents a probability that the test driver willhave a fatigue event (e.g. stop paying attention or even fall asleep).The more PVT data and other signals used to train the model the moreaccurate the estimations of the probability that the test driver willhave a fatigue event. The model may then be associated with the testdriver, for instance using an identifying code or other information, andsaved for later retrieval in the storage system 450.

FIG. 8 is an example flow diagram 800 for training a model forestimating a likelihood of a fatigue event for a person tasked withmonitoring a vehicle operating in an autonomous driving mode which maybe performed by one or more server computing devices, such as the servercomputing devices 410. At block 810, results from a plurality of sets ofpsychomotor vigilance test administered to the person at differentpoints in time are accessed. The results include response times for theperson. At block 820, scores for each of the sets of tests aredetermined based on the results. At block 830, information from a remotemonitoring system (or an in-vehicle monitoring system) identifyingestimations of the person's amount of fatigue at different times isaccessed. At block 840, whether the information indicates that theperson experienced one or more fatigue events is determined. At block850, the determined scores and the determination of whether theinformation indicates that the person experienced one or more fatigueevents are used to train a model individualized to the person such thatwhen data from a set of tests are input into the model, and the modeloutputs a value indicative of a likelihood of the person experiencing afatigue event. As noted above, the training may provide parameter valuesfor the model for the person. When using the parameter values, the modelmay output a value indicative of a likelihood of the person experiencinga fatigue event.

When and how often the testing system administers a test may varydepending upon the situation. For instance, a test may be initiated whena vehicle is stopped, such as at the beginning or the end of a testdriver's shift, immediately before or after the test driver goes on abreak, etc. In addition, the testing may be performed more often duringnight time shifts than day time shifts. To administer a test, thevehicle's computing devices may send a signal to the testing system whensuch situations occur. The rate at which tests are administered may behigher at the start of a shift or towards the end of a shift. In thisregard, the server computing devices may have access to informationabout the vehicle that each test driver is tasked with monitoring aswell as information about the start and end of each test driver'sshifts. In other instances, the aforementioned other signals, such as aremote assistance operator or automated monitoring system may determinethat the test driver is experiencing or is likely to experience afatigue event. The remote assistance operator or remote monitoringsystem may cause a signal to be sent to the testing system to administera test.

The testing system may test a test driver by administering a set oftrials and evaluating the test driver's response times for those trialsas described above. The response times as well as date and timeinformation for the trials of a test, shift information about therelative point in the test driver's shifts, as well as any of theadditional information discussed above (if available) may then be sentto and received by the remote server computing device.

The remote server computing device may then use the reaction times todetermine whether the test driver passed or failed each of the set oftrials of a test. In this context, failing (as discussed above) mayindicate that the test driver is not sufficiently alert to competentlymonitor the autonomous vehicle. The server computing device may alsodetermine a score for the trials of the set. To do so, the remote servercomputing device may compare the reactions times to a threshold amountof time and determine a ratio as described above.

The model associated with the particular test driver, which as notedabove, was trained using data from prior tests administered to that testdriver as well as other signals as discussed above may be identified.The results of the determinations of whether the test driver passed orfailed each PVT of the set of PVTs or alternatively, the response times,may be input into the model in order to determine a value representativeof a likelihood of a fatigue event. In other words, the ratio as well asthe date and time information, shift information, and any of theadditional information may be input into the model for that particulartest driver to assess whether that particular test driver is likely tohave a fatigue event by providing a value as indicated above. Analyzingthe data at the server computing devices 410 may provide flexibility toquickly change logic, thresholds, etc., and also to allow remoteassistance operators, the server computing devices, or others to requirea given test driver to take another test as soon as possible (e.g. at anext break). In addition, the model can be sensitive to recent data,aggregated at the server computing devices 410 (or storage system 450).For example, if the most recent data from the remote monitoring system280 also suggests fatigue, the same level of test performance (e.g.score) may result in a higher probability of there being a fatigueevent.

In order to determine and initiate an intervention response, the valuemay then be compared by the server computing devices 410 to a thresholdvalue or plurality of threshold values to determine whether and whattype of intervention response to take. The threshold values or pluralityof threshold values may be hand-tuned or selected depending uponprecision and recall values desired for the PVT system. In other words,the thresholds may be lower (e.g. lower values) in order to increase theamount or magnitude of intervention responses or higher (e.g. highervalues) in order to decrease the amount or magnitude of interventionresponses. Once an intervention response is identified, the servercomputing devices may send a signal to the computing devices of thevehicle, the PVT system, and/or a remote assistance operator in order toinitiate the identified intervention response.

Various different types of intervention responses may be employed. Forinstance, intervention responses may include providing with supportiveoptions and, if applicable, task-reassignment. For example, the testdriver may be provided with another test or another set of trials, maybe connected with one of the remote assistance operators who caninteract with the test driver as discussed above, or may even berelieved of the duty of monitoring a vehicle for a current shift. Inthis regard, the tests and the model can be used by the server computingdevices 410 to automatically determine whether a test driver should betask-reassigned before or even during a shift. In some instances,depending on the test driver's overall test performance, the test drivermay be automatically or discretionarily assigned to less safety-criticaltask or the vehicle may even prevent the test driver from controllingthe vehicle in the manual driving mode based on the signal. For example,the test driver may also be assigned to tasks associated with improvedalertness, such as rest periods or tasks with higher engagement. Incases where a test driver has some indicators of fatigue but is stillbelow a lower value threshold as described above, additional in-car orremote monitoring may be added to improve fatigue detection should afatigue event occur.

The scoring for a particular test driver (which may be used as input tothe model) may also be adjusted based on past performance and/orphysical fatigue. For example, some test drivers may have fasterreflexes even when they are tired, and other test drivers may experienceincreases in response time due to physical fatigue. In this regard,these test drivers may have the same reaction time, but differentresponse times. As such, the threshold amount of time used to determinewhether a test driver passed or failed a particular trial may beadjusted. In order to do so, a model of how this test driver orsimilarly situated test drivers (same time of day, same part of a shift,etc.) reaction times change over time may be generated. The model may bea statistical regression model which provides another way to separateperformance variance due to fatigue from variance that is due to otherfactors. In another example, although each trial of a PVT is very shortin duration (e.g. merely lifting a finger), it does require physicalmovement which over time will vary in response time due to accumulatingfatigue (i.e. the muscles of the test driver's finger will becomefatigued, even if the test driver has not). Thus, over time, thethreshold amount of time used to determine whether a test driver haspassed or failed a trial may be increased according to the fluctuationsin the model.

In addition, depending upon the orientation of the touchscreen, the testmay be more or less prone to physical fatigue. For instance, if thetouchscreen is oriented in an upright position, the test driver may bemore prone to muscle fatigue because he or she must hold his or her armin an elevated position. As another example, a driver with a longerreach may incur physical fatigue faster than one with a shorter reach,who may compensate by leaning toward the touch-sensitive display orother user input device used to administer the trials. This, data mayalso be included in the model for determining the threshold amount oftime.

Similarly, depending on the hand-dominance of the test driver, variancemay differ as the physical posture differs for left-handed persons andright-handed persons. In this regard, there may be different models fordetermining the threshold amount of time for left-handed persons versusright-handed persons. Again, the left-handed person or right-handedperson models may be specific to this test driver, similarly situatedtest drivers (same time of day, same part of a shift, etc.), or all testdrivers.

FIG. 9 is an example flow diagram 900 for assessing a likelihood of aperson experiencing a fatigue event when the person tasked withmonitoring a vehicle operating in an autonomous driving mode which maybe performed by one or more server computing devices, such as the one ormore server computing devices 410. For instance, at block 910, a set ofresponse times for a psychomotor vigilance test administered to theperson is received. The test includes a plurality of trials. At block920, whether the person passed or failed each trial of the set of trialsis determined. At block 930, a model associated with the person isidentified. The model was trained using data from prior psychomotorvigilance tests administered to the person. At block 940, results of thedeterminations of whether the person passed or failed each trial of theset of trials are input into the model in order to determine a valuerepresentative of a likelihood of a fatigue event. At block 950, anintervention response is initiated based on the value.

The features described herein may provide for a reliable and effectivesystem for identifying possible fatigue events in persons tasked withmonitoring the driving of a vehicle operating in an autonomous drivingmode. For instance, because the model is trained specific to aparticular person or a similarly situated person, the model may be morelikely to identify outlier situations. For example, if a particularperson has an average score of 0.8 at a particular time of day over thelast three months, but then at one time, that same person scores a 0.7which may be one or more standard deviations outside of the normalperformance for that person, the model may still indicate that theperson is likely to have a fatigue event even though a score of 0.7 mayhave been considered acceptable under a more generalized standard.Moreover, because the threshold amount of time used to determine whethera person has passed or failed a particular PVT can be adjusted toaccommodate for the particular person or similarly situated persons,this may further increase the effectiveness of the system reduce falsepositives. In other words, threshold amounts of time may be set in a waythat accounts for individual variance, for instance, due to musclefatigue.

Unless otherwise stated, the foregoing alternative examples are notmutually exclusive, but may be implemented in various combinations toachieve unique advantages. As these and other variations andcombinations of the features discussed above can be utilized withoutdeparting from the subject matter defined by the claims, the foregoingdescription of the embodiments should be taken by way of illustrationrather than by way of limitation of the subject matter defined by theclaims. In addition, the provision of the examples described herein, aswell as clauses phrased as “such as,” “including” and the like, shouldnot be interpreted as limiting the subject matter of the claims to thespecific examples; rather, the examples are intended to illustrate onlyone of many possible embodiments. Further, the same reference numbers indifferent drawings can identify the same or similar elements.

The invention claimed is:
 1. A method of assessing a likelihood of aperson experiencing a fatigue event, the person tasked with monitoring avehicle operating in an autonomous driving mode, the method comprising:receiving, by one or more server computing devices, a set of responsetimes for a set of trials of a psychomotor vigilance test (PVT)administered to the person; determining, by the one or more servercomputing devices, whether the person passed or failed each trial of theset of trials; identifying, by the one or more server computing devices,a PVT model associated with the person from a plurality of PVT modelsassociated with a plurality of respective persons, wherein eachrespective PVT model of the plurality of PVT models is trained usingdata from prior PVTs administered to the respective person; inputting,by the one or more server computing devices, results of thedeterminations of whether the person passed or failed each trial of theset of trials into the identified PVT model in order to determine avalue representative of a likelihood of a fatigue event; and initiating,by the one or more server computing devices, an intervention responsebased on the value.
 2. The method of claim 1, further comprisingdetermining the intervention response by comparing the value to one ormore threshold values.
 3. The method of claim 1, further comprisingdetermining a score for the PVT based on the determinations of whetherthe person passed or failed each trial of the set of trials, wherein theresults include the score.
 4. The method of claim 3, wherein the scorerepresents a passing or failing rate for the person.
 5. The method ofclaim 1, further comprising inputting date and time information for theset of trials into the identified PVT model in order to determine thevalue.
 6. The method of claim 1, further comprising inputting shiftinformation about a relative point in time for a shift for monitoringthe vehicle of the person for the PVT into the identified PVT model inorder to determine the value.
 7. The method of claim 1, furthercomprising inputting an amount of time since a last break of the personfor the PVT into the identified PVT model in order to determine thevalue.
 8. The method of claim 1, further comprising inputtinginformation identifying where the person is with respect to his or hercircadian rhythm, for the PVT into the identified PVT model in order todetermine the value.
 9. The method of claim 1, further comprisingdetermining a threshold amount of time based on whether the person isleft-handed or right-handed, and wherein determining whether the personpassed or failed each trial of the set of trials is based on thethreshold amount of time.
 10. The method of claim 1, further comprisingdetermining a threshold amount of time using a model of how responsetimes change over time due to muscle fatigue, and wherein determiningwhether the person passed or failed each trial of the set of trials isbased on the threshold amount of time.
 11. The method of claim 1,further comprising determining a threshold amount of time using a modelof how response times change over time for the person, and whereindetermining whether the person passed or failed each trial of the set oftrials is based on the threshold amount of time.
 12. The method of claim1, further comprising determining a threshold amount of time using amodel of how response times change over time for a similarly situatedperson, and wherein determining whether the person passed or failed eachtrial of the set of trials is based on the threshold amount of time. 13.The method of claim 1, further comprising determining a threshold amountof time is based on an orientation of a user input device used toadminister the set of trials, and wherein determining whether the personpassed or failed each trial of the set of trials is based on thethreshold amount of time.
 14. The method of claim 1, wherein initiatingthe intervention response comprises sending one or more signals to oneor more computing devices of the vehicle, a PVT system that administeredthe PVT, and a computing device of a remote assistance operator.
 15. Themethod of claim 14, wherein the one or more signals include instructionsfor the PVT system to administer a second PVT.
 16. The method of claim15, wherein the one or more signals include instructions for the vehicleto stop operating in the autonomous driving mode.
 17. The method ofclaim 16, wherein the one or more signals include instructions for thevehicle to prevent the person from operating the vehicle in a manualdriving mode.
 18. The method of claim 15, wherein the one or moresignals enable the person to communicate with the remote assistanceoperator.
 19. The method of claim 1, wherein the one or more servercomputing devices are further configured to determine a threshold amountof time using a model of how response times change over time for theperson, and wherein the one or more server computing devices are furtherconfigured to determine whether the person passed or failed each trialof the set of trials further based on the threshold amount of time. 20.A system for assessing a likelihood of a person experiencing a fatigueevent, the person tasked with monitoring a vehicle operating in anautonomous driving mode, the system comprising one or more servercomputing devices configured to: receive, a set of response times for aset of trials of a psychomotor vigilance test (PVT) administered to theperson; determine, whether the person passed or failed each trial of theset of trials; identify a PVT model associated with the person from aplurality of PVT models associated with a plurality of respectivepersons, wherein each respective PVT model of the plurality is trainedusing data from prior PVTs administered to the respective person; inputresults of the determination of whether the person passed or failed eachtrial of the set of trials into the identified PVT model in order todetermine a value representative of a likelihood of a fatigue event; andinitiate an intervention response based on the value.
 21. The system ofclaim 20, further comprising the vehicle.
 22. The system of claim 20,wherein the one or more server computing devices are further configuredto determine a threshold amount of time based on whether the person isleft-handed or right-handed, and wherein the one or more servercomputing devices are configured to determine whether the person passedor failed each trial of the set of trials further based on the thresholdamount of time.